Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services)

Thanks for working on this @Kavindu-Dodan !!
@axw Do you have opinion on the naming for ec2 tags here from the processor?

Tags are added to event payload with key ec2.tag.<KEY>,
{
  "ec2" : {
   "tag" : {
      "org" : "acme",
      "owner" : "userA"
    }
   }
}

Just a reference: In AWS CloudWatch metricset, we are using aws.tags.<key>:<value>.

@kaiyan-sheng my first reaction was that I think the tags field should be independent of the AWS service, because you can tag all/most AWS resources and not just EC2 instances. That could be useful generally for filtering metrics & logs by organisation or something.

OTOH, opentelemetry-collector-contrib's resourcedetection processor uses ec2.tag.*: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/40fa8b8a925cadf569e785cbc85d6dfca152bde2/processor/resourcedetectionprocessor/internal/aws/ec2/ec2.go#L29 (maybe that's what you modelled after @Kavindu-Dodan?)

OTOOH: it seems that resourcedetectionprocessor only supports adding tags for EC2, e.g. there's no support for tags for ECS: open-telemetry/opentelemetry-collector-contrib#14960.

So... I don't know, but I'd be inclined to go with aws.tags.<key>:<value>

@kaiyan-sheng my first reaction was that I think the tags field should be independent of the AWS service, because you can tag all/most AWS resources and not just EC2 instances. That could be useful generally for filtering metrics & logs by organisation or something.

OTOH, opentelemetry-collector-contrib's resourcedetection processor uses ec2.tag.*: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/40fa8b8a925cadf569e785cbc85d6dfca152bde2/processor/resourcedetectionprocessor/internal/aws/ec2/ec2.go#L29 (maybe that's what you modelled after @Kavindu-Dodan?)

OTOOH: it seems that resourcedetectionprocessor only supports adding tags for EC2, e.g. there's no support for tags for ECS: open-telemetry/opentelemetry-collector-contrib#14960.

Yes @axw , I came across OTel resource detection processor EC2 tag [1] and tried to match our implementation with that. Currently processor also support Azure tags [2]

So... I don't know, but I'd be inclined to go with aws.tags.<key>:<value>

Since we already use aws.tags. with AWS Cloudwatch metrics and to have consistency, I will go ahead and use the same format for the processor implementation.

Update

I am testing the tag prefix change and will update the observations. Initially I am seeing some data ingestion issue but not sure this is related to tag key prefix.

Update - 2

Okay, it was a permission issue when starting metricsbeat inside my AWS ec2 testbed. Change works fine (see Kibana screenshot)

@kaiyan-sheng @axw appreciate another review

[1] - https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.112.0/processor/resourcedetectionprocessor/internal/aws/ec2/ec2.go#L29
[2] - https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.112.0/processor/resourcedetectionprocessor/internal/azure/azure.go#L23